Reconfiguration of MPLS/WDM Networks Using Simulation-based Markov Decision Processes

We present a new approach to the problem of logical topology reconfiguration in congested WDM networks using simulation-based multi-time scale Markov Decision Processes (MDP). Previous work considered total reconfiguration of the network, branch exchanges or addition and deletion of lightpaths when the traffic pattern changes. However, the special case when all or part of the network is congested has not been studied in detail before. We demonstrate that our approach performs well in both uncongested and congested regimes. We restrict the actions taken in reconfiguring the network to branch exchanges to limit the effect of reconfigurations on the number of dropped calls. At each time step a list of possible branch exchanges is formed, and using a heuristic algorithm this (usually long) list is trimmed down to a more manageable list. We use the MDP model to predict the traffic, and then a simulation based online algorithm is used to select one of the branch exchanges as the next action. We apply a policy improvement method to the heuristic policy to obtain a better policy than the heuristic policy.